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To  assess  the  effectiveness  of  short-term  training  programs  for  senior 
national  security  officials,  research  was  conducted  on  two  fronts.  First 
there  was  the  research  that  was  required  to  develop  a program  that  could  serve 
as  a suitable  model,  i.e.  the  new  Harvard  Executive  Program  in  National  Security. 
Second,  was  that  involved  in  assessing  how  effectively  the  program  met  its 
basic  objectives. 

The  research  to  support  the  program  itself  primarily  involved  the 
development  of  appropriate  case~~studies . As  the  program’s  principal  pedagogical 
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instrument,  the  case  study  represents  one  of  the  more  rewarding  forms  of  re- 
search, particularly  where  application  is  concerned.  Exposure  to  high-level 
officials  provides  an  immediate  and  demanding  test  of  accuracy  and  effective- 
ness and,  through  the  feedback  that  ensues,  serves  to  enrich  the  quality 
of  the  case  materials. 

Since  the  program  participants  are  experienced  observers  of  events  and 
well-qualified  to  quantify  their  impressions  (Flag  and  General  rank  military 
and  civilians  of  comparable  status),  each  who  attended  the  first  year’s  session 
was  asked  to  evaluate  the  standard  parameters  of  program  assessment  and  to 
assess  the  degree  to  which  something  useful  had  been  learned  in  each  of  nine 
major  areas  of  emphasis.  The  following  represents  their  evaluation  of  the 
,Tstandard  parameters”  — on  a scale  of  1 (ineffective)  to  5 (extremely  effective) : 


Overall  usefulness 

4.31 

Instructors 

4.06 

Cases 

4.00 

Discussion  groups 

3.69 

Administration 

4.78 

The  principal  objective  of  the  program  was  to  enhance  the  participants’ 
understanding  of  differences  in  interest,  perspective,  and  style  associated 
with  varying  degrees  of  responsibilities  in  differing  organizations  (and  to 
thereby  enhance  their  ability  to  devise  and  implement  effective  solutions) . 

The  overall  score  on  meeting  this  objective  was  4.20.  In  refining  this  aspect 
of  the  assessment,  each  participant  was  asked  to  evaluate  on  a scale  of  100  to 
1000  the  degree  to  which  he  or  she  felt  competent  to  deal  with  issues  in  each 
of  the  five  prescribed  areas  — prior  to  the  program,  immediately  following 
the  program,  and  six  months  later  (after  sufficient  time  had  elapsed  to  per- 
mit meaningful  application  of  what  had  been  learned) . In  this  regard,  the 
average  participant  entered  the  program  with  a mean  "competence"  of  619  across 
all  areas.  During  the  course  of  the  program,  this  index  increased  to  714,  a 
fifteen  percent  improvement* 

Although  the  intent  of  the  program  was  to  provide  tools  which  the 
participants  would  find  useful  on  an  ongoing  basis,  it  was  nevertheless  an- 
ticipated that  the  index  would  fall  off  somewhat  during  the  six  months  following; 
completion  of  the  program  (as  any  post-program  euphoria  dissipated) . This 
proved  not  to  be  the  case,  however,  as  the  index  continued  upward  another  four 
percent  to  the  741  level.  It  would  appear,  then,  that  the  effort  to  provide 
"tools"  succeeded  beyond  original  expectations. 

In  comparing  the  performance  of  those  who  had  previously  received  War 
College  training  with  those  who  had  not,  it  was  noted  that  the  former  gained 
more  from  the  program  (both  during  and  after) , even  though  they  entered 
with  high  levels  of  felt  competence. 
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Effectiveness  of  Short  Term  Training 
Programs  for  Senior  National  Security  Officials 

To  assess  the  effectiveness  of  short-term  training  programs 
for  senior  national  security  officials,  research  is  being  conducted  on  two 
fronts.  First  there  is  the  research  that  is  required  to  develop  a program 
that  can  serve  as  a suitable  model,  i.e.  the  new  Harvard  Executive  Program 
in  National  Security.  Second  is  that  which  is  required  to  assess  how  ef- 
fectively the  program  meets  its  basic  objectives. 

The  research  to  support  the  program  itself  primarily  involves 
the  development  of  appropriate  case  studies.  As  the  program’s  principal 
pedagogical  instrument,  the  case  study  represents  one  of  the  more  reward- 
ing forms  of  research,  particularly  where  application  is  concerned.  Ex- 
posure to  high-level  officials  provides  an  immediate  and  demanding  test 
of  accuracy  and  effectiveness  and,  through  the  feedback  that  ensues, 
serves  to  enrich  the  quality  of  the  case  materials.  This  certainly 
proved  to  be  the  case  with  the  six  new  cases  that  were  developed  for  the 
first  session  of  the  Executive  Program  in  National  Security.  These  in- 
cluded two  cases  on  the  Panama  Canal  Treaty  negotiations,  one  on  the 
Philippine  Base  negotiations,  and  three  on  Eurocommunism. 

While  the  basic  supporting  research  is  a critical  ingredient  in 
the  makeup  of  the  program,  it  is  the  assessment  of  program  effectiveness 
that  constitutes  the  principal  focus  of  this  report.  In  determining  the 
best  approach  for  evaluating  program  effectiveness,  a thorough  knowledge 
of  the  background  leading  to  the  program’s  inception  becomes  helpful. 
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BACKGROUND 

As  the  panoply  of  government  problems  has  grown  increasingly 
complex,  the  need  for  improved  executive  development  in  the  public  sec- 
tor has  become  more  readily  apparent.  The  very  manner  in  which  the 
Federal  Service  is  structured,  with  its  continual  turnover  of  political 
appointees  in  most  of  the  top  policy  jobs  (some  of  whom  are  qualified; 
others  not)  tends  to  foster  inefficiency  in  policy  design  and  execution. 

A recent  survey  of  top  level  officials  within  the  national 
security  community  suggests  a strong  need  to  improve  management  capabil- 
ity in  both  the  military  and  civilian  spheres.  On  the  military  side, 
there  is  currently  very  little  available  in  the  way  of  continuing  educa- 
tion at  the  "corporate  executive  level."  While  there  is  a plethora  of 
training  programs  and  opportunities  available  for  military  personnel  at 
the  0-4  through  0-6  levels  (Lieutenant  Commander  through  Captain;  Major 
through  Colonel),  there  is  very  little  beyond  that.  Each  of  the  Services 
does  send  its  Flag  officer  selectees  to  an  indoctrination  course  of  two 
or  three  weeks  duration,  but  these  courses,  for  the  most  part,  have  an 
internal  Service  focus  and  tend  to  be  somewhat  mechanical  in  nature. 

Although  little  in  the  way  of  executive  training  is  offered  for 
Flag  and  General  officers,  the  need  for  something  substantial  is  quite  real. 
In  some  respects,  the  term  "General"  ought  to  equate  to  "generalist 
manager."  In  most  instances,  however,  the  path  to  Flag  or  General  officer 
rank  is  quite  specialized;  and  once  there,  the  individual  can  serve  for 
as  many  as  ten  to  fifteen  years  with  virtually  no  opportunity  for  intel- 
lectual or  managerial  refurbishment.  Adding  to  the  problem  is  the  fact 
that  there  are  certain  areas  of  critical  importance  to  the  top  Service 
manager  which  are  not  covered  well  in  any  of  the  multitudinous  forums  to 
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which  he  or  she  may  have  been  exposed  along  the  way.  Here  reference  is 
specifically  made  to  how  business  is  conducted  between  the  Department 
of  Defense  (DOD)  and  the  Congress;  how  the  different  perspectives  of  the 
Office  of  the  Secretary  of  Defense  (OSD),  the  Joint  Chiefs  of  Staff  (JCS) , 
and  the  Services  come  into  play;  what  is  involved/required  in  getting  deci- 
sions implemented;  and  the  like.  The  situation  is  further  exacerbated  by 
the  inevitable  tendency  for  the  incumbent  (or  the  incumbent Ts  superior) 
to  feel  that  the  job  demands  at  that  level  are  too  consuming  to  permit 
extrication  for  any  significant  period  of  training. 

The  needs  on  the  civilian  side  are  even  more  pressing.  In 
May  of  1974,  the  U.  S.  Army  Management  Engineering  Training  Agency  (AMETA) 
conducted  a "Study  of  Management  and  Executive  Development  in  Industry, 
Universities,  and  the  Federal  Government."  In  this  study,  the  authors 
struck  a comparison  between  executive  development  efforts  in  the  private 
and  public  sectors  and  found  the  latter  wanting.  The/ basically  concluded 
that  executive  development  in  government  is  anything  but  institutionalized; 
that  there  is  little,  if  any,  accountability  by  senior  managers;  and  that 
there  is  no  system  for  identifying,  training,  and  tracking  prospective 
executives . 

More  recently  in  March  of  1976,  a panel  of  the  National  Academy 
of  Public  Administration  completed  a study  entitled  "Strengthening 
Civilian  Executive  Development  in  the  Department  of  Defense."  Conducted 
at  the  request  of  the  Assistant  Secretary  of  Defense  (Manpower  and  Reserve 
Affairs) , the  study  generated  a number  of  recommendations  that  accommodated 
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what  the  panel  found  at  that  time  to  be  the  general  characteristics  of 
Federal  Civil  Service  (i.e.,  limited  lateral  entry,  little  mobility  be- 
tween agencies  or  occupational  groups  and  virtually  no  "conscious  system" 
for  developing  executive  leadership)  while  specifically  addressing  the 
unique  problems  of  DOD.  In  this  regard,  the  panel  saw  the  need  for  con- 
sistent top  level  support  of  executive  development  programs  as  the  most 
critical  problem  facing  the  Department.  Sheer  size  coupled  with  the  fact 
that  most  career  posts  in  Defense  are  occupied  either  by  political  ap- 
pointees or  the  military  (with  attendant  brief  periods  of  incumbency)  makes 
it  very  difficult  to  personify  the  top  level  interest  that  is  required. 

The  National  Academy  study  also  pointed  out  that  over  the  five 
year  period  from  1976  through  1981,  approximately  50%  of  the  DOD  super- 
grade workforce  would  become  eligible  for  retirement.  Thus,  it  was  sug- 
gested that  a primary  focus  of  DOD  executive  training  should  be  to  develop 
the  replacements  for  what  is  likely  to  be  an  inordinately  high  number  of 
vacancies.  Related  to  this  suggestion  was  the  fact  that  the  civilian 
personnel  system,  while  not  a closed  system,  is  heavily  dominated  by  single 
career  personnel.  Indeed,  approximately  80  percent  of  DOD  GS  16-18  jobs 
are  filled  from  within  the  Department*  The  inference  is  twofold:  (1)  it 
is  unrealistic  to  shortchange  executive  development  on  the  basis  that 
potential  candidates  have  already  received  such  training  in  private  indus- 
try  prior  to  entering  government  and  (2)  any  executive  development  program 
for  DOD  should  be  heavily  slanted  toward  meeting  the  specific  requirements 
associated  with  managing  the  national  security  process. 
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In  view  of  the  above.  Harvard  University  decided  in  April  of 
1978  to  offer  a two  week  Executive  Program  in  National  Security  on  an 
annual  basis,  with  the  first  program  to  take  place  in  August  of  the  same 
year.  In  January  1979,  the  Navy  Personnel  Research  and  Development  Cen- 
ter published  a Navy  Civilian  Executive  Study  that  implicitly  endorsed 
the  intent  behind  the  Harvard  effort  (and  any  others  like  it) : 

Although  relatively  few  executives  have  had  extensive 
academic  training  in  management,  leadership,  or  administration, 
they  spend  most  of  their  time  performing  tasks  in  these  areas. 

This  highlights  a major  training  need.  It  includes.  . .general 
management  knowledge  and  skills  (e.g.  decision-making,  communi- 
cations) . . .A  need  was  also  identified  for  the  integration  of 
civilian  and  military  training  in  the  shore  establishment. 

COURSE  DESCRIPTION 


Modeled  after  earlier  programs  pioneered  by  the  Business  School 
for  senior  business  executives,  the  Executive  Program  in  National  Security 
(subseqently  renamed  the  Executive  Program  in  National  and  International 
Security)  is  designed  for  Flag  and  General  rank  military  officers  and  for 
high-level  civilian  officials  who  are  either  in,  or  moving  into,  posts  where 
their  personal  decisions  or  recommendations  can  critically  affect  the  poli- 
tical, economic,  or  military  interests  of  the  United  States.  Its  principal 
goals  are  to  improve  participants'  understanding  of: 

— differences  in  interest,  perspective  and  style  associated 

with  varying  types  of  responsibilities  in  differing  organizations 

— the  economic,  political,  technological,  and  organizational  con- 
text within  which  national  security  policies  and  programs  are  framed 

— various  dimensions  of  high-level  management,  including  the  uses 
and  misxises  of  formal  analysis  in  decision-making  and 

policy  design 
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COURSE  EVALUATION 


Since  the  participants  are  experienced  observers  of  events  and 


well-qualified  to  quantify  their  impressions,  each  who  attended  the  first 
year’s  session  was  asked  to  evaluate  what  might  be  considered  the  standard 


parameters  of  program  assessment  (Attachment  A) . The  following  summarizes 
the  more  salient  of  these: 


1.  On  a scale  of  1 (ineffective)  to  5 (extremely  effective): 


a. 

Overall  usefulness 

4.31 

b. 

Instructors 

4.06 

c. 

Cases 

4.00 

d. 

Discussion  groups 

3.69 

e. 

Administration 

4.78 

2.  With  respect  to: 


Program  expectations  Work  load 


Program  length 


20 — equalled  29 — about  right 

8 — exceeded  1 — too  heavy 

1 — fell  short  1 — too  light 

1 — parts  exceeded; 
parts  fell  short 


29 — about  right 
1 — too  long 
1 — too  short 


3.  Comparison  of  pedagogical  approach  with  those  experienced  in 
other  educational  programs:  4.22 


While  the  principal  themes  of  the  program  include  management 
perspective,  policy  development,  and  uses  of  analysis,  it  is  the  former 
that  constitutes  the  central  thread  of  the  curriculum.  In  getting  par- 
ticipants to  think  critically  about  the  perspectives,  stakes,  and  interests 
of  their  counterparts  in  and  around  government,  it  becomes  necessary  to 
analyze  the  setting  in  which  problems  arise  and  the  particular  incentives 
which  various  actors  face.  Individuals  behave  differently  in  different 
organizational  settings  according  to:  (1)  their  own  history  and  that  of 
their  organization,  (2)  the  reward /sanction  structure  posed  by  their 
career  system,  and  (3)  which  of  the  many  faces  of  an  issue  they  confront. 
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The  dimensions  along  which  these  aspects  are  treated  during  the  course 
of  the  program  include:  the  Congress,  political  appointees,  careerists, 
the  media,  business,  and  labor.  In  addition,  special  attention  was  also 
paid  to  budgeteers  and  analysts,  ethics  and  personnel  systems. 

The  overall  score  assigned  the  program  with  respect  to  meeting 
its  principal  objective  of  increasing  the  students  understanding  of  the 
perspectives  of  other  participants  in  the  national  security  process  was 
4.20.  In  refining  this  aspect  of  the  assessment,  a "ratio  scaling"  pro- 
cedure was  used  wherein  each  executive  was  asked  to  evaluate  on  a scale 
of  100  to  1000  the  degree  to  which  he  or  she  felt  competent  to  deal  with 
issues  in  each  of  the  above-cited  areas  of  emphasis  (Attachment  B) . In 
this  regard,  a series  of  three  data  points  were  taken:  one  prior  to  the 
program,  one  immediately  following  completion  of  the  program,  and  one 
six  months  later  (after  sufficient  time  had  elapsed  to  permit  meaningful 
application  of  what  was  learned.)  In  view  of  the  heavily  subjective  nature 
of  the  total  evaluation  scheme,  however,  the  pre-program  data  was  taken  at 
the  same  time  as  that  immediately  following  the  program.  This  was  done  to 
help  ensure  greater  consistency  in  standards,  i.e.,  the  participants  may 
not  have  fully  appreciated  at  the  outset  how  weak  (or  strong)  they  were 
in  a given  area  as  they  were  after  they  had  completed  the  program.  It 
also  had  the  added  benefit  of  avoiding  an  artificial  "ceiling  effect" 
wherein  one  might  have  given  oneself  the  highest  rating  possible  at  the 
beginning  of  the  program  and,  after  learning  considerably  more  during  the 
program,  then  been  forced  to  use  the  same  (but  subsequently  misleading) 
rating . 
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As  indicated  in  Table  I,  the  average  participant  entered  the 
program  with  a mean  "competence11  of  619  across  all  areas,  i.e.,  the 
"average  participant"  felt  about  62  percent  as  competent  or  comfortable 
in  an  "average  area"  as  he  or  she  did  in  an  area  with  which  they  were 
thoroughly  familiar  and  in  which  they  felt  well  qualified  (see  Attach- 
ments C and  D) . During  the  course  of  the  program,  this  mean  index  in- 
creased to  714,  a fifteen  percent  improvement  over  the  pre-program  level. 


Average  "Competence"  Across  All  Areas 

Pre-Program  Post-Program  Six  Months  Post 
Total  Group  (N=29)  619  714  741 

Although  the  intent  of  the  program  was  to  provide  tools  which 
the  participants  would  find  useful  on  an  ongoing  basis,  it  was  neverthe- 
less anticipated  that  the  index  would  fall  off  somewhat  during  the  six 
months  following  completion  of  the  program  (as  any  post-program  euphoria 
dissipated).  This  proved  not  to  be  the  case,  however,  as  the  index  con- 
tinued upward  another  four  percent  to  the  741  level.  It  would  appear,  then, 
that  the  effort  to  provide  "tools”  succeeded  beyond  original  expectations. 

Although  the  foregoing  provides  some  feel  for  short-term  program 
effectiveness,  even  more  revealing  are  the  increases  in  the  indices  for 
each  of  the  specific  areas  of  emphasis: 


Area 

TABLE  II 
Pre-program 
Level 

Gain  During 
Program 

Gain  in  Six  Months 
Following  Program 

Ethics 

762 

57 

16 

Personnel  Systems 

703 

61 

12 

Military  vs.  Career  Civilians 

699 

46 

22 

Budge teers /Analysts 

616 

100 

36 

Congress 

615 

105 

38 

Politicians 

612 

111 

30 

Business  and  Businessmen 

601 

96 

32 

Media 

589 

88 

28 

Labor  Relations 

376 

193 

24 
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In  general,  the  program  raised  the  perceived  level  of  competence 
in  all  areas  but  had  little  effect  on  the  overall  rank  order  (Budgeteers/ 
Analysts,  Congress,  and  Politicians  reversed  positions,  going  from  616,  615, 
and  612  to  716,  720,  and  723  respectively).  For  Labor  Relations,  the  area 
with  the  lowest  initial  level  of  felt  competence,  a rather  dramatic  in- 
crease from  376  to  569  was  realized.  However,  at  the  end  of  the  program 
the  respondents  still  had  less  confidence  in  their  ability  to  operate  in 
this  area  than  in  any  other.  With  one  exception,  the  respondents  learned 
most  in  those  areas  where  they  initially  knew  the  least.  The  exception, 
as  indicated  in  Table  II,  was  in  the  "media"  category  where  a number  of 
respondents  indicated  lower  competence  both  immediately  following  the  pro- 
gram and  six  months  later.  The  reason  behind  this  has  to  do  with  the  na- 
ture of  the  program  presentation  made  in  this  area.  In  exploring  the  nuances 
of  the  First  Amendment,  it  illustrated  (and  quite  effectively)  the  non- 
monolithic  character  of  the  press.  Hence,  participants  left  the  experience 
feeling,  at  least  in  some  instances,  that  they  didnTt  know  as  much  about  the 
media  as  they  originally  thought  they  did  upon  first  entering  the  program. 

In  comparing  the  reactions  of  those  participants  who  had  at- 
tended War  College  with  those  who  had  not,  it  was  noted  that  the  former 
entered  the  program  feeling  more  competent  (647  vs.  594)  and  left  feeling 
that  they  had  gained  more  (103  vs.  91  during  the  program,  and  31  vs.  25 
after) . Thus  it  appears  that  the  War  College  graduates  generally  gained 
more  from  the  program  than  did  their  non-War  College  peers  (see  Attachment  E) . 
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As  part  of  the  six  month  follow-on  assessment,  participants 
were  asked  to  what  extent  their  improved  understanding  of  the  perspectives 
of  others  had  helped  them  in  the  performance  of  their  jobs,  particularly 
as  manifested  in:  (1)  the  development  of  goals  and  the  strategy  to  sup- 
port achievement  of  those  goals,  and  (2)  the  framing  and  presentation  of  programs 
in  mixed  arenas.  They  rated  these  two  categories  at  3.96  and  3.80  res- 
pectively. 

THE  LONGER  VIEW 

Since  the  six  months  data  shows  a continuing  increase  in  levels 
of  competence/ comfort , the  question  arises  as  to  what  one  might  expect  to 
see  with  respect  to  long  term  program  impact,  e.g.,  a year  following  pro- 
gram completion.  The  analysis  below  examines  as  one  possibility  the 
existence  of  a Markov  relationship  between  successive  six  month  data  points 
and  contrasts  the  results  with  those  that  would  obtain  from  simple  asympto- 
tic extrapolation. 

To  explore  the  possibility  of  an  underlying  stochastic  process, 
transition  matrices  were  constructed  for  the  pre-program  (Event  1)  vs. 
immediate  post  program  (Event  2)  case  and  for  the  immediate  post  program 
vs.  six  month  post  program  (Event  3)  case  within  each  "area  of  compe- 
tence." The  first  step  in  doing  so  required  determination  of  the  median 
level  of  competence  for  the  29  sets  of  pre-program  data  associated  with 
each  "area"  (three  participants  did  not  provide  complete  sets  of  data) . 

The  next  step  was  to  determine  the  + .25  quantile  points  (upper  and  lower 
"hinges")  which  could  serve  as  the  boundaries  for  three  discrete  "zones 
of  competence"  (See  Attachment  F) . The  Event  1-2  matrix  for  each  area 
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was  then  compared  with  the  Event  2-3  to  determine  whether  any  differences 
that  existed  were  statistically  significant.  Table  III  shows  the  calcula- 
tions for  the  "Business"  area. 


TABLE  III 


"Business"  Event  1-2  and  Event  2-3 
Transition  Matrices 


EVENT  1-2 

H ( > 800) 

M (575  - 800) 

L ( < 575) 


00 

0 

0 

5 

9 

o 

0 

4 

2 

EVENT  2-3 


13 

0 

0 

2 

11 

0 

0 

1 

2 

Applying  a Chi  Square  test  for  goodness  of  fit: 

•'  ^ >\  Zft  i „ ± . ± 

£ 2 3 H 


3 


--  1M 


(where  Event  1-2  is  the  "EXPECTED"  result  and  Event  2-3  is  the  "OBSERVED") 


It  is  apparent  from  the  Chi  Square  test  that  if  the  Event  1-2/Event  2-3 
relationship  were  stationary,  one  would  expect  to  see  differences  of  the 
magnitude  of  those  shown  above  less  than  10  percent  of  the  time.  Since 
the  findings  are  similar  in  each  of  the  other  eight  areas  (see  Attach- 
ment G) , it  is  apparent  that  the  differences  are  statistically  signifi- 
cant. This  suggests  one  of  two  possibilities:  either  the  relationship 
is  not  Markov,  or  it  is  time  dependent  and  has  not  yet  reached  its  sta- 
tionary state  (if  one  does  indeed  exist) . 
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To  explore  the  possibility  of  a non-stationary  Markov,  it  will 
be  necessary  to  collect  yet  another  set  of  data  at  Time  4,  i.e.  one  year 
following  completion  of  the  program.  In  anticipating  what  that  data 
collection  might  yield,  one  approach  that  suggests  itself  is  to  assume 
continued  operation  of  the  Event  2-3  dynamics  and  to  apply  the  Event  2-3 
transition  matrix  to  the  State  3 distribution  to  yield  a new  "state  dis- 
tribution" for  Time  4.  Making  these  calculations  for  the  "Business" 
category: 

H M L 


] 15  12  2~] 

L 

X 

i 

0 

0 

17 

.15 

.85 

0 

- 

11 

0 

.33 

.67 

1 

^Distribution  x Event  2-3  Transi-  = Expected  T, 

tion  Matrix 

Thus,  if  the  assumption  holds  true  that  the  same  Event  2—3  dynamics  are 
at  work  during  Event  3-4,  cne  might  expect  to  find  17  participants  having 
evolved  to  the  high  state,  11  to  the  medium,  and  1 to  the  low. 

To  compare  possible  trends  in  the  absence  of  T^  data,  simple 
asymptotic  extrapolation  was  performed  using  the  following  equation: 

=>  X,  [i  * .23^CX^)] 

where  the  constant  ,284  is  obtained  by  dividing  the  increase  in  mean  level 
of  competence  over  the  six  months  following  the  program  (741-714)  by  the 
increase  that  took  place  during  the  program  (714—619) • Application  of 
this  technique  to  each  of  the  29  participants T Time  3 scores  in  the 
Business  area  yields  a Time  4 distribution  of  19  in  the  high  state,  8 in 
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the  medium,  and  2 in  the  low.  Table  IV  compares  the  results  of  applying 
the  T2_3approach  simple  asymptotic  extrapolation  to  the  data  associated 
with  each  of  the  nine  areas  of  emphasis. 


TABLE  IV 

AREA  APPLY  EVENT  ASYMPTOTIC 

2-3  MATRIX  EXTRAPOLATION 


H 

M 

L 

H 

M 

L 

1. 

Congress 

21 

8 

0 

21 

7 

1 

2. 

Politicians 

19 

10 

0 

17 

10 

2 

3. 

Media 

13 

15 

1 

14 

13 

2 

4. 

Budget 

22 

6 

1 

22 

5 

2 

5. 

Bus iness 

17 

11 

1 

19 

8 

2 

6 . 

Mil /Civ 

16 

10 

3 

14 

12 

3 

7. 

Personnel 

9 

18 

2 

13 

12 

4 

8. 

Labor 

17 

12 

0 

16 

13 

0 

9. 

Ethics 

9 

20 

0 

11 

18 

0 

As  shown  in  the  table,  continued  application  of  the  Event  2-3  matrix  yields 
results  which  are  generally  quite  close  to  those  determined  through  extra- 
polation . 

As  stated  earlier,  a data  collection  at  Time  4 will  be  required 
to  determine  which,  if  either,  of  the  above  projections  reflects  the  actual 
distribution  that  will  obtain  at  that  time.  However,  if  one  accepts  the 
premise  that  the  program  has,  in  fact,  provided  "tools"  that  will  be  of 
continuing  use,  the  possibility  of  a Markov  relationship  between  successive 
six  month  data  points  (probably  displaying  Event  2-3  dynamics)  cannot  be 


discounted. 
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P REDICTION  OF  APPLICANTS 

Yet  another,  albeit  less  direct,  measure  of  effectiveness  is 
the  degree  to  which  successive  runs  of  the  program  attract  increasing 
numbers  of  applicants.  Since  there  is  only  a single  year’s  worth  of  data 
currently  available,  it  is  not  possible  at  this  point  to  predict  with  any 
certainty  the  likely  number  of  applicants  for  succeeding  years.  It  is 
possible,  however,  to  explore  the  construction  of  a hypothetical  model 
which  can  be  refined  with  successive  runs  of  the  program. 

The  first  step  in  developing  such  a model  would  be  to  define  the 
relationship  between  the  number  of  applicants  from  each  agency  and  any 
variables  that  seem  important.  In  this  case,  the  variables  might  include 
the  number  of  eligible  candidates  within  the  agency  (both  military  and 
civilian),  the  agency’s  training  budget  (dollars  available  per  eligible 
candidate),  the  number  of  program  brochures  sent  to  the  agency,  and  the 
number  of  personal  marketing  visits  paid  to  individuals  within  that  agency. 
To  determine  the  relationship,  a multiple  regression  model  has  been  used. 

In  this  type  of  model,  an  attempt  is  made  to  derive  the  equation  of  the 
following  form  that  best  fits  the  observed  data: 

^■L  = (5e,  ^ (3,  y ,i  *•  (3>2  Xz'<  f ^ 

where  £/jl  = no.  of  applicants  from  agency 

X»l  = training  budget  for  agency 
Xcl  = no.  of  eligible  candidates  from  agency 
Xzl  =no.  of  brochures  mailed  to  agency 

Xm  = no.  of  visits  to  agency 
= regression  coefficients 

In  other  words,  the  attempt  is  made  to  find  the  linear  combination  of 
independent  variables  that  best  fits  the  observed  dependent  variable.  The 
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”best  fit”  is  determined  by  choosing  the  (3  coefficients  so  as  to  minimize 

A 

the  resultant  squared  error  of  the  estimates.  If  is  the  estimated 

A -j-  A 

vector  of  coefficients,  and  ^ > - fa  X j then  (h  is  chosen  so  that 
Z-  14)  J - J is  a minimum  over  all  possible  choices  of 

In  this  case,  a packaged  stepwise  multiple  linear  regression 


program  was  used,  and  analysis  was  performed  using  the  data  in  Table  V. 

It  should  be  noted  that  this  is  a biased  sample:  it  does  not  include  data 
from  agencies  which  sent  no  attendees.  Therefore,  the  derived  equation 
does  not  provide  estimates  of  how  many  applicants  will  come  from  a given 
agency  with  a given  budget,  etc.,  but  rather,  estimates  of  the  number  of 
applicants,  assuming  that  there  was  at  least  one  applicant  in  FY  1978. 


TABLE  V 

Regression  Data  1978 


Agency 

//Appli- 

cants 

//Eligible 

Civ  Mil 

Training 
Budget  ($ 
per  person 

Bro- 

chures 

Visits 

Army 

1 

297 

432 

652 

25 

4 

Navy 

2 

372 

261 

790 

41 

7 

Air  Force 

6 

193 

269 

1045 

23 

3 

Marine  Corps 

1 

5 

66 

390 

5 

2 

Naval  Reserve 

1 

0 

40 

307 

3 

2 

Army  Reserve 

1 

0 

92 

420 

3 

2 

OSD/JCS 

4 

346 

0 

112 

64 

5 

Coast  Guard 

1 

0 

31 

303 

8 

1 

State 

1 

1047 

0 

435 

39 

7 

AID 

1 

344 

0 

334 

10 

1 

Treasury 

1 

314 

0 

274 

4 

1 

0MB 

1 

80 

0 

63 

5 

1 

*Series  60  (level  66)/6000  Time  Sharing  Application  Library  Guide,  Vol.  II 
Statistics;  Stepwise  Linear  Regression  Program.  Honeywell  Information 
Systems,  Inc.;  copyright  1976. 
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In  proceeding  on  a stepwise  basis,  it  was  determined  that  training 
budget  was  the  most  significant  variable  for  a one  variable  equation.  For 
a two  variable,  "brochures"  was  the  next  most  significant;  for  a three,  the 
number  of  personal  visits.  No  equation  with  four  variables  provided  any 
significant  additional  "explanation"  of  the  variance  in  the  number  of  appli- 
cants, The  results  of  the  analysis  were  as  follows: 

//applicants  = ,42  + .004  (budget)  + .105  (//brochures)  - .790  (//visits) 

This  equation  was  found  to  explain  almost  75%  of  the  variance  in  the  depend- 
ent variable.  Conspicuously  absent  is  the  variable  relating  to  number  of 
eligible  participants,  since  it  did  not  provide  sufficient  information  to 
merit  its  inclusion.  Also  of  note  is  the  negative  coefficient  relating  to 
numbers  of  personal  visits.  As  with  any  regression,  care  should  be  taken 
not  to  draw  causal  inference.  In  this  case,  it  should  not  be  concluded 
that  the  visits  adversely  affected  the  number  of  applicants.  It  was, 
instead,  a matter  of  concentrating  attention  on  those  agencies  which  were 
most  resistant  to  sending  anyone  in  the  first  instance. 

It  was  originally  hoped  that  the  number  of  1979  applicants  could 
be  estimated  using  this  regression  equation  since  the  estimated  FY79  training 
budgets  are  known  for  each  agency,  as  are  the  projected  numbers  of  visits 
and  brochures.  However,  the  visitation  and  brochure  policies  have  changed 
significantly  between  1978  and  1979.  In  FY78  only  one  agency  received  more 
than  fifty  brochures  and  all  but  four  received  at  least  two  visits.  In 
FY79  four  agencies  received  well  over  100  brochures  and  only  two  received 
more  than  1 visit.  This  change  in  the  tone  of  the  recruiting  effort  would 
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render  any  estimate  of  applicants  highly  unreliable.  When  the  FY79 
applicants  are  known,  the  enlarged  sample  should  permit  the  formulation 
of  a more  comprehensive  regression  model.  The  most  complete  statement 
that  can  be  made  at  this  time  is  that,  in  FY78,  the  relationship  among 
the  dependent  and  independent  variables  was  as  given  above. 

The  second  component  of  the  model  looks  at  the  application 
process  as  a random  Poisson  process  within  each  agency.  The  Poisson 
process  is  characterized  by  an  event  with  a small  probability  of  occur- 
rence, but  a large  number  of  possible  occurrences.  In  this  situation,  the 
probability  that  a specific  individual  from  any  given  agency  will  attend 
the  program  is  small, , yet  there  are  many  such  individuals  who  are 
eligible  to  attend.  Although  there  is  insufficient  information  in  a 
single  sample  to  determine  whether  this  process  does,  indeed,  fit  the 
Poisson  model  (this  can  only  be  tested  after  several  samples  have  been 
taken),  it  is  not  unreasonable  to  assume  a fit  at  this  point  in  time. 

Assuming  a Poisson  process,  then,  the  expected  number  of  appli- 
cants from  agency  l is  Ai.  The  probability  of  having  X applicants 
from  an  agency  that  has  mean  is  given  by: 

-a: 

P ((/applicants  = x)  = ^ * 

X t 

This  second  part  of  the  model  connects  with  the  first  through  substitution 
of  A't  as  the  dependent  variable,  iji  , and  regressing  Al  on  the  recruiting 
variables.  In  this  formulation,  it  is  the  mean  number  of  applicants  AL 
that  is  expressed  as  a function  of  the  recruiting  variables,  while  the 
actual  number  of  applicants  in  a given  year  is  a Poisson  random  variate 
with  mean  A;  . The  probability  of  the  actual  number  equalling  X may 
then  be  determined  from  the  above  equation. 

*Note  that  X is  usually  expressed  as  a rate  in  a At  combination.  In  this 

situation,  \ has  already  been  normalized  and  t falls  out. 
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The  third  part  of  the  model  links  the  Poisson  models  for  the 
individual  agencies  into  a complete  model  for  the  whole  program.  If  the 
\i  were  equal  for  all  agencies,  the  predicted  total  number  of  applicants 
would  merely  be  the  sum  of  the  predicted  number  from  each  agency.  How- 
ever, if  the  A l are  not  equal,  then  the  situation  would  be  heterogeneous 
and  the  predicted  number  from  each  agency  would  have  to  be  weighted  by  the 
variance  of  the  individual  estimates.  The  total  number  of  applications 
then  is  a function  of  the  individual  At  and  some  process  which  samples 
from  the  various  Poisson  processes.  A prime  candidate  for  this  sampling 
process  is  the  Gamma  distribution.  In  other  words,  the  total  number  of 
applicants  would  be  a random  variable  that  is  the  sum  of  individual 
Poisson  processes  sampled  according  to  a Gamma  distribution.  If  this  is 
the  case,  then  the  total  number  can  be  shown  to  follow  a negative  binomial 

distribution.  Putting  the  whole  model  together: 

A JL  _ 

A » 

1.  Ai  - ^ ( Ntis  linear  in  the  recruiting  variables) 

2.  Prob  ( //  atteiu  = k)  = ^ £ (each  agency  i has  k applicants 

k i 

according  to  a Poisson  process  with  mean  \l  ) 


3.  X t itself  comes  from  a Gamma  distribution:  -r  L\) 


resulting  in  a Negative  Binomial 

Prob  (k  = n)  = qVpn  (-l)n  ("X) 

where  p = — and  q = 1 - p. 

I 

The  mean  /a  = v p/q  and  variance  = M-C)  + 


V -I 


-a.  A 


if  A >o 
; f y\  - o 
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The  next  step  is  to  test  this  negative  binomial  model  by  applying 
it  to  the  national  security  program  data. 

* 


1.  Computed  = 21  applicants 

17  agencies  ** 


= 1.24 


2.  Computed  variance  = ^ (# applicants  in  agency  i - 1.24)  __  9 , 

n-1 

where  n = 17 


3.  Using  the  identities  for  the  mean  and  variance  of  the 

negative  binomial,  parameters  "a"  and  ‘'v~for  this 

distribution  can  be  computed  using  the  sample  mean 

and  sample  variance  values  of  1.24  and  2.3  respectively. 

° s ^ , 

V | p ■ i 

° 5 'nc<£L  p - ~ 4 - i - p r - 1 ’ yr 

1 i <>  r 6 i i > ^ 

. _ . 

6 , . V - { „ 2^0- 

• no^ce  (c3-)  r 2-5  - /*0+^)  = i-^(  ' + 

• V.  IMS  ; - = U7  , P-.£( 

4.  Substituting  the  above  values  into  the  model: 

f-  yro  lr  ( h - <'»)  - ^VpM(~0  C »-t ) 

= C*‘>A0,'^C^O"  r 0‘H±*»)  ^ - Hi 

r?  I PI.4S  -?<1  ^TT~ 

- ( *H4>)  (.,*-/  (j)  ! ^ + rl) 

ri  i 


Eight  of  the  twenty-nine  applicants  came  from  organizations  which  do 
not  lend  themselves  to  analysis,  e.g.,  intelligence  agencies  whose 
training  budgets,  etc.  are  classified. 

The  "17  agencies"  figure  includes  five  agencies  which  yielded  no 
applicants . 
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5,  The  probabilities  for  different  numbers  of  applicants  can  now  be  determined 


K 

0 

1 

2 

3 or  more 


Prob  n = K 

• 41 

• 27 

• 15 

• 17 


6,  Performing  a Chi  Square  test,  we  find  the  following: 

# candidates 

0 1 2 

Expected  //  agencies  (as  7 5 3 

generated  by  negative 
binomial) 

Actual  # agencies  59  1 

2 4 16  4 

X =y  + + y = -57  + 3,2  + 1.33  + .33 


3 or  more 
3 


2 


X3DF  - 5,4  ^ one  would  expect  to  see  differences  on  the  above  order 

approximately  15%  of  the  time 

While  the  possibility  that  the  number  of  agencies  sending  k applicants  is 
distributed  as  a Negative  Binomial  cannot  be  rejected,  it  may  be  useful  to 
test  a Poisson  distribution.  A glance  at  the  respective  distribution 
curves  reveals  why. 

Frequency 
of  K 

0 1 2 3 0 1 3 3 0 1 2 3 

National  Security  Negative  Binomial  Poisson 

Program 
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Applying  a Poisson  model  yields  the  following: 

Pro  U-  ( l<  i ^ A 

n * 

Kor  > ^ 1.2^  , Pro/,  (k-Ki)  = .Z?3  A" 

r?  i 


Table  VI  compares  the  results  obtained 

from  both  models 

with  actual  program 

experience. 

TABLE  VI 

Probability 

that  n = k 

K Negative 

Binomial 

Poisson 

Observed 

0 .41 

.289 

.294 

1 .27 

.359 

.529 

2 .15 

.222 

.059 

3 or  more  .17 

.12 

.118 

A Chi  Square  test  of  the  fit  of  the  data  to  Poisson  distribution  with  mean 
1.24  is  approximately  3.44  with  three  degrees  of  freedom,  so  the  hypothesis 
that  the  number  of  agencies  sending  k applicants  is  distributed  as  a 
Poisson  (where  k = o,  1,  2..oo)  cannot  be  rejected. 

Although  additional  samplings  will  be  required  to  determine  with 
precision  what  the  correct  distribution  actually  is,  from  the  calculations' 
above,  it  seems  most  likely  that  the  distribution  will  prove  to  be  Poisson. 

If  so,  the  assumption  of  heterogeneity  will  have  been  disproved.  While  there 
are  obviously  a number  of  unquantif iable  factors  at  play  within  each  agency 
that  influence  the  number  of  participants  the  agency  ultimately  sponsors 
(such  as  internal  politics  and  the  like) , it  may  hold  true  that  these  fac- 
tors are  similar  enough  across  agencies  so  as  to  largely  offset  one  another. 
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As  stated  earlier,  data  limitations  have  dictated  that  the  re- 
search relating  to  assessment  of  long-term  effectiveness  assume  the  form 
of  a prospectus.  It  is  only  with  the  additional  data  that  will  come  from 
successive  runs  of  the  program  that  the  validity  of  the  models  can  be  con- 
firmed — or  the  need  to  refine  or  revise  them  verified. 


Attachment  A 


HARVARD  UNIVERSITY 

EXECUTIVE  PROGRAM  IN  NATIONAL  SECURITY 
August  13-25,  1973 

Program  Evaluation  Questionnaire 

Please  complete  and  return  to  Doug  Johnston  no  later  than  Friday  morning,  Aug.  25. 

Your  candid  responses  regarding  the  following  aspects  of  the  Executive  Program 
in  National  Security  will  assist  us  in  assessing  the  effectiveness  of  the  program 
and  improving  its  quality  and  value  for  the  future. 

Note : In  those  questions  calling  for  a quantified  evaluation,  please  use  the 

following  5-point  scale: 

1 = Ineffective  (or  Unsatisfactory) 

2 « Below  average  effectiveness  (or  only  Marginally  Satisfactory) 

3 ■ Average  effectiveness  (or  Satisfactory) 

4 »*  Above  average  effectiveness  (or  Very  Good) 

5 = Extremely  effective  (or  Excellent) 

A.  Overall  Evaluation  of  the  Program 

1.  Please  evaluate  the  Program  in  terms  of  overall  usefulness  to  you. 

Ineffective  12345  Extremely  Effective 

2.  Evaluate  the  usefulness  of  Discussion  Groups. 

Ineffective  12345  Extremely  Effective 

3.  How  did  the  Program  compare  with  your  expectations  from  the  announce- 
ment you  received? 

Exceeded  Expectations 

Equaled  Expectations 

Fell  Short  of  Expectations 


Comments : 
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4.  How  would  you  assess  the  work  load? 

Too  Heavy 

_____  About  Right 

5.  How  would  you  describe  the  length  of  the  program? 

Too  Long 

_____  About  Right 
Too  Short 

6.  If  you  do  not  think  the  program  was  the  right  length,  vhat  length 
would  you  suggest? 

1 week 

3 weeks 

Longer 

/.  Would  you  attend  if  it  were  3 weeks?  yes no (Why?) 

8.  Is  there  a more  preferable  period  than  August? 

Concent : 


Evaluation  of  Classes 


1.  Please  indicate  your  evaluation  of  the  cases  and  lecture-discussions 
for  class.  Please  comment  if  you  think  any  individual  classes  were 
particularly  effective  or  ineffective. 

INDOCHINA  *54 

Ineffective  12345  Extremely  Effective 

CUBAN  MISSILE  CRISIS 

Ineffective  12345  Extremely  Effective 

TAXING  INDUSTRIAL  DEVELOPMENT  BONDS 

Ineffective  12345  Extremely  Effective 
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INTERNAT IONAL  ECONOMICS  I & II 


Ineffective  1 2 

DEFENSE  BUDGEr 

Ineffective  1 2 

PRINCIPLES  OF  NEGOTIATION: 

3 4 5 

3 4 5 

DISTRIBUTIVE 

Extremely  Effective 

Extremely  Effective 

BARGAINING 

Ineffective 

L 

2 

3 

4 

5 

Extremely  Effective 

EMPLOYMENT  SERVICE  REORGANIZATION 

Ineffective 

1 

2 

3 

4 

5 

Extremely  Effective 

ASPIN  AND  DEFENSE  BUDGE!  CUTS 

Ineffective 

1 

2 

3 

4 

5 

Extremely  Effective 

BAY  OF  PIGS 

Ineffective 

1 

2 

3 

4 

5 

Extremely  Effective 

THE  MEDIA  AND  NATIONAL  SECURITY 

Ineffective 

1 

2 

3 

4 

5 

Extremely  Effective 

INTEGRATIVE  BARGAIN IN 

G (AMPO -ADMINISTRATION  NEGOTIATION) 

Ineffective 

1 

2 

3 

4 

5 

Extremely  Effective 

PANAMA  CANAL  TREATY 

Ineffective 

1 

2 

3 

4 

5 

Extremely  Effective 

PHILIPPINE  EASE  NEGOT 

IATIONS 

Ineffective 

1 

2 

3 

4 

5 

Extremely  Effective 

KOREA  1980 

Inetfect ive 

1 

2 

3 

4 

5 

Extremely  Effective 

ENERGY  AND  NATIONAL  SECURITY  I . 

1 1, 

III 

Ineffective 

1 

2 

3 

4 

5 

Extremely  Effective 

SATURDAY  NIGHT  MASSACRE 

Ineffective 

1 

2 

3 

5 

5 

Extremely  Effective 
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FORCE  POSTURE  I,  II 

Ineffective  1234 

NUCLEAR  ENERGY  AND  PROLIFERATION 

5 

Extremely  Effective 

Ineffective 

1 

2 

3 

4 

5 

Extremely  Effective 

STRATEGIC  FORCES  IN 

1960s 

Ineffective 

1 

2 

3 

4 

5 

Extremely  Effective 

SALT  I 

Ineffective 

1 

2 

3 

4 

5 

Extremely  Effective 

AMERICAN  INDUSTRIES 

CORP  (A 

i B, 

C) 

Ineffective 

1 

2 

3 

4 

5 

Extremely  Effective 

U .S . -USSR  GRAIN  AGREEMENT 

Ineffective 

1 

2 

3 

4 

5 

Extremely  Effective 

MARX- ENGELS  TEXTS 

Ineffective 

1 

2 

3 

4 

5 

Extremely  Effective 

SALT  II 

Inefiect ive 

1 

2 

3 

4 

5 

Extremely  Effective 

EUROCOMMUNISM 

I & 11 

Ineffective 

1 

2 

3 

4 

5 

Extremely  Effective 

EUROCOMMUNISM 

EXERCISE 

Ineffective 

1 

2 

3 

4 

5 

Extremely  Effective 

2.  How  would  you  compare  Che  effectiveness  of  the  teaching  methods 
employed  in  this  program  with  others  you  may  have  experienced  in 
ocher  educational  programs? 

Ineffective  12345  Extremely  Effective 

(Please  comment) 
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3.  In  comparison  with  the  quality  of  teaching  skills  personally 
observed  in  other  seminars  and  courses,  please  indicate  your 
evaluation  of  the  in-class  performance  of  each  faculty  mem- 
ber in  this  program. 

GRAHAM  ALLISON 


Ineffective  1 

ALBERT  CARNESALE 

2 

3 

5 

5 

Extremely  Effective 

Ineffective  1 

JOHN  DUNLOP 

2 

3 

4 

5 

Extremely  Effective 

Ineffective  1 

STEPHEN  HITCHNER 

2 

3 

4 

5 

Extremely  Effective 

Ineffective  1 

WILLIAM  KAUFMANN' 

2 

3 

4 

5 

Extremely  Effective 

Ineffective  1 

ERNEST  MAY 

2 

3 

4 

5 

l 

Extremely  Effective 

1 

Ineffective  1 

ARTHUR  MILLER 

2 

3 

4 

5 

Extremely  Effective 

Ineffective  1 

JONATHAN  MOORE 

2 

3 

4 

5 

Extremely  Effective 

Ineffective  1 

HOWARD  RAIFFA 

2 

3 

4 

5 

Extremely  Effective 

Ineffective  1 

HUGO  UYTERHOEVEN 

2 

3 

4 

5 

Extremely  Effective 

Ineffective  1 

DANIEL  YERGIN 

2 

3 

4 

5 

Extremely  Effective 

Ineffective  1 

2 

3 

4 

5 

Extremely  Effective 
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C.  Program  Content 

1.  In  what  areas  would  you  have  liked  to  have  seen  greater 
emphasis  placed? 


2.  Was  anything  omitted  that  should  have  been  included? 


3.  What  suggestions  would  you  make  for  improving  the  design  of  the  Program 


D.  Participation 

1.  Was  the  mix  of  the  participants  appropriate  in  terms  of: 

No  Yes 

Organizational  Level  _____  _____ 

Experience  _____  

Ability  to  Contribute  in  Class  _____  ____ 

Ability  to  Contribute  to  Discussion  Groups  ___^  


Comments: 
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2.  Did  you  feel  you  had  sufficient  opportunity  to  contribute  in  class- 
room discussions? 

3.  How  did  you  first  learn  about  the  Executive  Program  in  National  Security? 

Direct  correspondence  from  Program  Director  

Another  person  in  your  organization  - (colleague, 
personnel  department,  other  - please  specify)  

Program  brochure  

JFK  or  HBS  Faculty  member  

Other  (please  specify)  

E.  Administration 

1.  Please  evaluate  the  level  of  administrative  organization. 

Poor  12345  Excellent 

2.  Please  evaluate  the  flexibility  and  responsiveness  of  the  administration. 

Poor  12345  Excellent 

3.  How  would  you  evaluate  the  living  accommodations? 

Poor  12345  Excellent 

4.  How  would  you  evaluate  the  lounge  facilities? 

Poor  12345  Excellent 

5.  How  would  you  evaluate  other  HBS  facilities  which  you  used? 

Poor  1 2 3 4 5 Excellent 

6.  How  would  you  evaluate  the  meals? 

Poor  12345  Excellent 

* 

/.  Comments  and  suggestions: 


-8 


F.  Additional 


1.  What  difference,  if  any,  has  the  program  made  in  your  perception 
of  your  own  role  in  the  national  security  process? 


2.  How  effective  has  the  program  been  in  increasing  your  understanding 
of  the  perspectives  of  other  participants  in  the  national  security 
process? 

Ineffective  12345  Extremely  Effective 


3.  (Answer  only  at  the  six  months  point) 

If  the  program  has  improved  ycur  understanding  of  the  perspectives 
of  others,  to  what  extent  has  this  understanding  subsequently 
proven  helpful  in  performing  your  job,  particularly  with  respect 
to: 

(a)  development  of  goals  and  the  strategy  to  support 
achievement  of  those  goals: 

Ineffective  12345  Extremely  Effective 

(b)  framing  and  presenting  programs  in  mixed  arenas 
Ineffective  12345  Extremely  effective 


S ignature 


Do  you  think  participation  from  the  private  sector  would  enhance  or  detract 
from  the  program? 


Would  participants  from  foreign  countries  inhibit  the  flow  of  discussion'f 
On  balance  do  you  thinic  their  presence  would  enhance  or  detract  from  the 
program? 


Attachment  3 


Ratio  scale  ratings  relating  to 
the  perspectives  and  roles  of  other 
participants  in  the  national  security  process 


Please  evaluate  how  comfortable  you  feel  in  dealing  with/opera- 
ting in  each  of  the  listed  areas.  If  you  feel  completely  comfortable 
about  operating  in  an  area  with  which  you  are  very  familiar,  a rating  of 
1,000  would  be  appropriate.  If  you  only  feel  "half  comfortable,"  then 
the  rating  becomes  500.  If  you  feel  one-tenth  as  comfortable  (as  com- 
pared to  a very  well-known  area),  then  the  rating  should  be  100. 

Ratings  of  zero  (0)  and  negative  ratings  should  not  be  used. 

The  questions  listed  within  each  area  are  intended  to  be  repre- 
sentative of  the  types  of  things  you  should  consider  in  deciding  how  com- 
fortable you  are.  They  should  not  be  interpreted  as  representing  an  ex- 
haustive treatment  of  any  given  area. 


|£l  fU 


RATIO  SCALE  RATINGS 


or  ORE  AFTnR  6 MOS 
RCGRAM  PRO  GUAM  LATER 


I.  Congress 

- Wnat  are  the  nuances  of  the  legislative  process  that  might 
make  a difference  in  the  way  one  frames  one’s  programs 
prior  to  submission  for  Congressional  approval? 

- what  are  the  priorities  that  are  likely  to  be  accorded  com- 
peting demands  for  resources  by  the  various  national 
security-related  Congressional  committees? 

- On  what  issues  is  Congressional  decision-making  likely  to  be 
driven  by  the  national  interest,  as  opposed  to  local  politica 
considerations? 

- How  should  one  deal  with  the  situation  where  it  looks  likely 
that  the  substance  of  the  issue  may  be  subsumed  In  juris- 
dictional infighting,  i.e.  , protection  of  turf  vis-a-vis 
the  Executive  Branch,  between  committees,  etc.? 

- To  what  extent  should  one  provide  one’s  personal  opinion 
vis-a-vis  unquestioning  support  for  the  Administration 
position? 


II.  Media 


- How  does  one  reconcile  the  Inherent  conflict  between  the 
media’s  interpretation  of  obligations  and  rights  under  the 
First  Amendment  and  the  need  for  secrecy  that  often  under- 
lies "national  security  considerations"? 

- What  motivates  the  press  in  its  pursuit  of  news? 

- How  can  one  use  the  media  to  help  achieve  one’s  own  objective 

- How  does  one  avoid  press  chicanery,  such  as  the  slanting  of 
stories  to  achieve  sensationalism  and  quoting  out  of  context? 

- What  is  the  language  with  which  one  should  be  familiar  when 
dealing  with  the  media  ("off  the  record,”  "deep  background," 
"non-attribution, " etc.)? 

+ 

- Is  citing  a "personal  opinion"  ever  appropriate  for  a top 
level  official  when  dealing  with  the  media? 
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III.  Politicians 


- How  should  one  deal  with  the  fact  that  the  terns  and  condi- 
tions of  employment  of  politicians  are  considerably  different 
from  those  for  other  categories  of  participants  in  the 
national  security  process? 

- What  motivates  them? 

- How  does  their  different  risk  orientation  affect  their  be- 
havior and  perceptions  of  things? 

- How  does  the  politician’s  role  compare  with  the  role  of  the 
Chief  Executive  Officer"  in  a business  organization? 

- What  is  the  best  way  to  cope  with  the  managerial  shortcomings 
of  the  various  categories  of  political  appointees?  i.e., 
educators  often  are  not  skilled  in  the  management  ot  large 
organizations;  lawyers  are  often  not  skilled  in  either 
quantitative  analysis  or  the  administration  of  large 
organizations;  and  businessmen  often  have  trouble  operating 
in  a "no  bottom  line"  environment. 

- In  what  sorts  of  circumstances  are  political  considerations 
likely  to  prevail  over  those  of  substance? 


IV.  Budgeteers /analysts 

- How  does  one  stay  ahead  of  analytical  chicanery? 

- How  can  one  be  sure  one  is  asking  the  right  questions  and 
forcing  the  right  options? 

- To  what  extent  is  it  desirable  or  necessary  to  instill  an 
appreciation  for,  and  at  least  limited  accommodation  of, 
organizational  process  and  bureaucratic  politics  amongst 
budgeteers  and  analysts? 


V.  Business  and  businessmen 


■ How  should  one  deal  with  businessmen  so  as  to  elicit  the 
most  forthcoming  response? 

- What  is  the  "business  perspective,"  and  the  incentives  that 
drive  it? 

- What  are  the  ramifications  of  the  fact  that  most  national 
security  program  managers  spend  more  of  their  time  "selling" 
their  programs  than  they  do  managing  them? 
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VI.  Military  vs.  career  civilians 

- What  are  the  differences  in  planning  priorities  between 
military  and  civilian  planners? 

- How  do  trade-offs  between  hardware  and  doctrine  influence 
weapons  acquisitions? 

- What  are  the  effects  of  frequent  turnover  of  military  person- 
nel on  program  decision-making  and  accountability? 

- To  what  extent  does  a "civil  service  mentality"  affect 
creativity  and  initiative  in  program  formulation  and 
implementation? 


VII.  Personnel  systems 

- What  are  the  incentives  for  inspiring  top  performance  in  both 
the  military  and  civilian  spheres? 

- How  do  you  make  the  personnel  system  work  for  you  instead  ot 
against  you  when  it  comes  to  hiring  and  firing  and  effecting 
changes  in  organizational  structure? 

- How  does  one  avoid  the  debilitating  effects  of  civil  servants 
working  for  military  bosses  who  often  are  unconcerned  with 
civilian  career  advancement,  etc.V 


VIII.  Labor  relations 

- How  does  one  deal  effectively  with  unions  in  the  public  secto 

- What  are  the  points  of  leverage  available  to  the  public  secto 
manager? 


IX.  Ethics 


- How  does  one  reconcile  the  inevitable  conflict  between 
principle  and  expediency,  particularly  when  being  "too 
truthful"  may  result  in  one  pricing  onself  out  of  the 
market  with  respect  to  procurement  of  a particular  weapon 
system  (which  in  the  eyes  of  the  sponsor  is  tritical  to 
the  national  interest)? 

- What  are  one's  moral  obligations  vnen  relating  with  the 
Congress?  the  press? 


Signature 
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